Deep Multimodal Guidance for Medical Image Classification

AbstractMedical imaging is a cornerstone of therapy and diagnosis in modern medicine. However, the choice modality for particular theranostic task typically involves trade-offs between feasibility using (e.g., short wait times, low cost, fast acquisition, reduced radiation/invasiveness) expected performance on clinical diagnostic accuracy, efficacy treatment planning guidance). In this work, we aim to apply knowledge learned from less feasible but better-performing (superior) guide utilization more-feasible yet under-performing (inferior) steer it towards improved performance. We focus application deep learning image-based diagnosis. develop light-weight guidance model that leverages latent representation superior modality, when training consumes only inferior modality. examine advantages our method context two applications: multi-task skin lesion classification dermoscopic images brain tumor multi-sequence magnetic resonance (MRI) histopathology images. For both these scenarios show boost without requiring Furthermore, case classification, outperforms trained while producing comparable results uses modalities during inference. make code models available at: https://github.com/mayurmallya/DeepGuide.KeywordsDeep learningMultimodal learningClassificationStudent-teacher learningKnowledge distillationSkin lesionsBrain tumors